{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Optimizer - SGD, Adam \n",
    "* @ python 3 matrix multiplication  \n",
    "* **Iterators** / **Generator** (Iter to make  -> next to grab \n",
    "Every generator is iterator, but not vice versa\n",
    "\n",
    "Iterator - any object whose class has __next__ method and __iter__ method.\n",
    "Generator - object build by calling a fn. that has `yield` in it (yield is like return but it remembers local variable even after exiting fn. and can be started from there with __next__ statement)\n",
    "\n",
    "-- can use `for o in `\n",
    "-- or `next()` --> stream processing (because after 1st, its NEXT (2nd) thing. not the 3rd thing  \n",
    "\n",
    "any dataset can be converted into iterator by wrapping in **DataLoader(d, shuffle = True, bs = 64)** {pytorch} (by default -- without replacement) \n",
    "\n",
    "* **Variable** i pytorch has similar api as tensor, but keeps track of what we did. Therefore can take derivative  \n",
    "    * `.backward` gives gradient (do it on `loss` to calculate it's lowest point) \n",
    "    * goes inside functions like `chain rule`\n",
    "\n",
    "* **Function_()** here _ does inplace = True  \n",
    "\n",
    "* **Optimizer** -- `a.data -= learning_rate * a.grad.data` . \n",
    "\n",
    "* **Momentum** -- keep track of \n",
    "\n",
    "* If we have layer inside the net, we don't need to 0 the gradient in SGD.  \n",
    " \n",
    "* Why we do RELU (non linear activation funtion) ? If not, we will end up with combination of multiple linear layers which is not useful as it is just 1 linear layer with different parameters . \n",
    "* Final non linear layer (**softmax** - want prob. of 1 class (so use for binary), **sigmoid** - for multiclass) \n",
    "* For hidden non linear layer, pretty much always use **RELU** or **reaky RELU** (close to 0)(found overtime) . \n",
    "\n",
    "* **Broadcasting** - \n",
    "the smaller array is “broadcast” across the larger array so that they have compatible shapes (smaller array means of lower rank) \n",
    "\n",
    "* **expand_dims** to change vector to 1 column matrix  \n",
    "* c[None] - adds 1 axis at start of len 1. c[:,None] - adds new axis at end of len 1 . c[None:,None] - adds new axis at start and end   \n",
    "* **backpropagation** - using chain rule to find derivates . \n",
    "\n",
    "* **Eliminate overfitting** -- reduce the weights or predict better are ways to minimize loss fn. Eg. `L1` or `L2` = sum of abs weights or sum of sq weights . Common values of it's {lambda} = 1e-6 to 1e-4 . \n",
    "\n",
    "derivate for `L2` -> 2a.sum(w) (w = weights). called: **weight decay** . used to reduce training overfitting  \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Broadcasting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the smaller array is “broadcast” across the larger array so that they have compatible shapes (smaller array means of lower rank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with rules of broadcasting --\n",
    "\n",
    "When operating on two arrays, Numpy/PyTorch compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward. Two dimensions are compatible when :\n",
    "\n",
    "* they are equal, or\n",
    "* one of them is 1  \n",
    "\n",
    "What does this mean? Let's see with examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Questions -- \n",
    "* Why used logsoftmax, not softmax?  \n",
    "* n.Parameter(torch.randn(*dims)/dims[0]) --> reason of dividing by dims[0]  \n",
    "* torch.log(torch.exp(x)/(torch.exp(x).sum(dim=0))) --> why non lin layer back to log. (we had aim to make +ves)  \n",
    "* "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HW  \n",
    "* Try adding non linear RELU"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
